System and method for ranking, allocation and pricing of content recommendations

ABSTRACT

A server computer system receives campaign data from one or more content providers. The server computer system determines assignments of recommendation opportunities in documents to recommendations based on the campaign data. The server computer system ranks the recommendations based on the assignments, allocates the recommendations to the documents based on the rankings of the recommendation opportunities, and determines pricing for the recommendations based on the allocations.

TECHNICAL FIELD

The present disclosure relates to content recommendations and, moreparticularly, to a technique of ranking, allocating and pricing contentrecommendations.

BACKGROUND

In the digital advertising industry, the onus of revenue-generatingactivity is shifting from selling interruptive and irrelevantadvertisements to providing integrated high-quality content that isinteresting, helpful and trustworthy to readers. The digitaladvertisement industry has not kept pace with this paradigm shift, anddigital solutions that natively support content marketing are lacking.Traditionally, content providers place bids to have their content placedas recommendations on various websites. In general, bidding platformspromise to allocate content to the highest bidder on animpression-by-impression basis. Some conventional bidding platforms useGeneralized Second Prize (GSP) to position the buyer bidding the highestfirst in the bidding queue. The buyer (e.g., content provider) thereuponpays the price suggested by the second highest bid. Traditional biddingplatforms do not offer means to determine balanced ranking, allocationand pricing of content recommendations.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will be understood more fully from the detaileddescription given below and from the accompanying drawings of variousimplementations of the disclosure.

FIG. 1 illustrates an example system architecture in which examples ofthe present disclosure can be implemented.

FIG. 2 is a block diagram of an optimization module, in accordance withvarious implementations.

FIG. 3 is a flow diagram of an implementation of a method for ranking,allocating and pricing recommendations.

FIG. 4 is a flow diagram of an implementation of a method for creatingan allocation plan.

FIG. 5 illustrates an example of creating an allocation plan, accordingto various implementations.

FIG. 6 is a flow diagram of one implementation of a method fordetermining the pricing for recommendations.

FIG. 7 is a block diagram of an example computer system that may performone or more of the operations described herein, in accordance withvarious implementations.

DETAILED DESCRIPTION

A system and method for ranking, allocating and pricing recommendationsis described, according to various implementations. Content providersmay wish to have their content distributed as recommendations on variousdestination documents. A content recommendation or recommendationhereinafter refers to a representation of digital content. Arecommendation can include one or more text, links, thumbnails, images,icons, etc. A document hereinafter refers to webpage, a mobileapplication graphical user interface (GUI), etc. For example, a webblogger, John Smith, may wish to have his web blog post of his analysisof a baseball game to be included as a recommendation for users that areaccessing a publisher's (for e.g. SportsNews, Inc.) webpage describingthe highlights of the same baseball game. An impression hereinafterrefers to an instance of populating a destination document (e.g.,webpage, mobile application GUI) with a content recommendation. Forexample, the web blog post of John Smith may be an impression if the webblog post is served as a recommendation on the SportsNews, Inc. webpage.

The content provider (e.g., web blogger John Smith) may use a thirdparty recommendation service provider to serve his content to variousdestination documents. The content provider can specify to the thirdparty recommendation server a budget and/or the cost-per-click (CPC) thecontent provider is willing to pay for the service. “Content provider”is hereinafter also referred to as “buyer.” The CPC is the amount thebuyer is willing to pay each time a content recommendation is selected.For example, the content provider may have a maximum budget of $800 andmay be willing to pay a maximum CPC of $1.

Implementations dynamically provide balanced estimation, ranking,allocation and pricing of content recommendations. Implementationsenable a variety of content recommendations to be served as impressionsby ranking the content recommendations, determining an optimalallocation of which content should be served as the impressions forvarious destinations documents (e.g., webpages, mobile applications userinterfaces, etc.) based on the rankings, and by determining an optimalprice a content provider should pay based on the allocations, whiletaking into account the parameters (e.g., budget, CPC) that arespecified by the content provider. Implementations can dynamicallydetermine the optimal ranking, allocation and pricing of recommendationsand can dynamically adjust the optimal ranking, allocation and pricingbased on real-time data.

FIG. 1 illustrates an example system architecture 100 in which examplesof the present disclosure can be implemented. The system architecture100 can include one or more user devices 160A-B, one or more contentprovider servers 101,103,105, one or more destination servers (e.g.,publishing servers 120) and one or more third party recommendationservers 140 coupled to each other over one or more networks 130.

Content providers may have recommendations (e.g., recommendations107,109,111) they wish to have displayed, for example, in a publisherdocument (e.g., document 150,160). A document 150,160 may be forexample, and is not limited to a website homepage, section front,webpage, mobile application user interface (UI), gaming application UI,television application UI, etc. For example, content provider webblogger John Smith may have multiple recommendations 107 of various webblog posts and may wish to have the recommendations 107 served indocument 150 (e.g., SportsNews webpage) and/or document 160 (e.g.,BaseballNews website). A recommendation 107,109,111 can be arepresentation of digital content. Digital content can include, forexample, such as, and not limited to, an article, a web blog entry, asocial media posting, a data feed, an advertisement, a document (e.g.,webpage), digital video, digital movies, digital photos, digital music,social media updates, electronic books (ebooks), electronic magazines,digital newspapers, digital audio books, electronic journals, electroniccomic books, software applications, etc. Each content provider can havemultiple recommendations 107,109,111.

A content recommendation host or “publisher” hereinafter refers to anentity and platform hosting destination documents (e.g., webpages,mobile application graphical user interfaces (GUIs), gaming applicationGUIs, television application GUIs, etc.) that include recommendations107,109,111 of the content providers. For example, SportsNews, Inc. andBaseballNews, Inc. may be publishers. The publisher platforms caninclude publishing servers 120. The publisher servers 120 can beapplication servers that provide one or more applications (e.g., webapplication, mobile application, desktop application, gaming consoleapplications, television console application, etc.). The publishingservers 120 can be coupled to one or more data stores that storepublisher data and/or recommendations of content providers.

The publishers' destination documents 150,160 can be rendered with therecommendations 107,109,111 of the content providers in user devices160A-B using for example, an application (e.g., web browser, mobileapplication, desktop application, gaming application, televisionapplication, etc.) on the user device 160A-B.

A document 150,160 can be within a source 147,157. The source 147,157can be for example, and is not limited to, a publisher's website, apublisher's mobile application, publisher's gaming application,publisher's television application, etc. A destination document 150,160can include publisher content (e.g., publisher content 153,163). Forexample, the SportsNews website (e.g., source 147) can include a webpage(e.g., document 150) that contains an article about a baseball game(e.g., publisher content 153).

The publisher destination documents 150,160 can be configured with oneor more recommendation opportunities 155,165, which can be populatedwith recommendations 107,109,111. For example, a document 150 (e.g.,SportsNews webpage) may have four recommendation opportunities 155 and adocument 160 (e.g. BaseballNews mobile application GUI) may have threerecommendation opportunities 165. The recommendation opportunities155,165 may be populated with recommendations 107,109,111 from one ormore content providers' servers 101,103,105.

The content providers may wish to publish their respectiverecommendations 107,109,111 in widespread locations (e.g., document 150,document 160). The content providers can use the service hosted by thethird party recommendation service provider to have the recommendations107,109,111 of the content providers served to populate therecommendation opportunities 155,165 in the publisher documents 150,160.The publishers (e.g., SportsNews Inc., BaseballNews, Inc.) can use theservice hosted by the third party recommendation service provider toreceive recommendations 107,109,111 to populate the recommendationopportunities 155,165 in their respective publisher documents 150,160.

The content providers may purchase campaigns from the third partyrecommendation service provider. The campaigns provide recommendations107,109,111 from the content providers to one or more destinationdocuments 150,160 in accordance with criteria selected by the contentprovider and parameters of the documents 150,160 set by the publishers.The destination documents 150,160 may have parameters defining the scopeof which recommendations 107,109,111 may be included in the destinationdocuments 150,160. For example, the parameters can include, and are notlimited to, category (e.g., news), sub-category (e.g., internationalnews), country, platform, language, etc.

The content providers may specify parameters for the campaign, such as,and not limited to, a maximum budget (B) of what the content provider iswilling to pay for the campaign and the maximum CPC that the contentprovider is willing to pay each time users (e.g., users 161A,B) select arecommendation.

The third party recommendation service provider can host one or morethird party recommendation servers 140 to provide the service to thecontent providers and publishers. The servers 140 can include anoptimization module 145 to determine an optimal allocation of whichcontent provider recommendations 107,109,111 should be served asimpressions to populate the recommendation opportunities 155,165 in thedocuments 150,160 while taking into account the campaign parameters, thedocument parameters, and the likelihood that the impressions would beselected by users 161A-B and provide high user-engagement whilegenerating click-through revenue. The publishers (e.g., SportsNews Inc.,BaseballNews, Inc.) may wish to include impressions in their respectivedocuments 150,160 that are of sufficiently high quality while providinga substantial income stream in return for hosting the recommendations.The optimization module 145 can select the recommendations 107,109,111to be served which meet the criteria of the content providers andpublishers. The optimization module 145 can select the recommendations107,109,111 that provide the most valued content while still providingsubstantial overall revenue and increasing user-engagement throughoutthe network.

The network of content providers can include, for example, severalhundred thousand recommendations 107,109,111 that may be stored in oneor more data stores that are coupled to the network 130. Thearchitecture 100 can include, for example, several hundred publisherdocuments 150,160 that are accessed by multiple users 161A-B on varioususer devices 160A-B.

For example, document 150 may have four recommendation opportunities155. When a particular user (e.g., user 161A) accesses the particulardocument 150, the optimization module 145 can determine which set ofrecommendations 107,109,111 should be served as impressions to populatethe four recommendation opportunities 155 in the document 150. As eachuser 161A-B accesses a document 150,160, the optimization module 145 candetermine which set of recommendations 107,109,111 should be served forthe particular user and document combination.

The click-through rate (CTR) is the estimated probability (calculated asa percentage) that a user 161A-B will select the recommendation if therecommendation populates a recommendation opportunity in a destinationdocument. The CTR can be determined from user activity data that isstored in one or more data stores that a coupled to the network 130.

The data stores can be persistent storage units coupled to the network130. A persistent storage unit can be a local storage unit or a remotestorage unit. Persistent storage units can be a magnetic storage unit,optical storage unit, solid state storage unit, electronic storage units(main memory), or similar storage unit. Persistent storage units can bea monolithic device or a distributed set of devices. A ‘set’, as usedherein, refers to any positive whole number of items.

The networks 130 can include one or more local area networks (LAN), oneor more wireless networks, one or more mobile communications networks,one or more wide area networks (WAN), such as the Internet, or similarcommunication systems, or combinations of such. The networks 130 caninclude one or more networking and/or computing devices such as wiredand wireless devices. In one implementation, not all servers101,103,105,120,140, and/or user devices 160A-B are directly connectedto each other. The networks 130 may be separate networks.

The servers 101,103,105,120,140 can be hosted on one or more machines.The machines can include for example, and are not limited to, any dataprocessing device, such as a desktop computer, a laptop computer, amainframe computer, a personal digital assistant, a server computer, ahandheld device or any other device configured to process data.

A user device 160A-B can be a computing device such as a servercomputer, a desktop computer, a set-top box, a gaming console, atelevision, a portable computing device such as, and not limited to,mobile telephones, personal digital assistants (PDAs), portable mediaplayers, netbooks, laptop computers, an electronic book reader and thelike. For example, user device 160A may be a laptop computer and userdevice 160B may be a tablet computer.

FIG. 2 is a block diagram of an optimization module 200, in accordancewith various implementations. The optimization module 200 can be thesame as the optimization module 145 in FIG. 1. The optimization module200 can include a user selection probability estimation sub-module 201,a real-time ranking, allocation and pricing sub-module 203, a masterplan builder sub-module 205, and an inventory estimation sub-module 207.Note that in alternative implementations, the functionality of one ormore of the user selection probability estimation sub-module 201, thereal-time allocation, ranking and pricing sub-module 203, the masterplan builder sub-module 205, and the inventory estimation sub-module 207are combined or divided. Note that in alternative implementations, thefunctionality of the optimization module 200 can be separate modules.

The optimization module 200 can be coupled to one or more data stores210,220,230,250,260,270,280 that store data. The data stores210,220,230,250,260,270,280 can be persistent storage units. Apersistent storage unit can be a local storage unit or a remote storageunit. Persistent storage units can be a magnetic storage unit, opticalstorage unit, solid-state storage unit, electronic storage units (mainmemory), or similar storage unit. Persistent storage units can be amonolithic device or a distributed set of devices. A ‘set’, as usedherein, refers to any positive whole number of items.

The data stores 210,220,230,250,260,270,280 can store input and/oroutput data. The modules (e.g., user selection probability estimationsub-module 201, the real-time ranking, allocation and pricing sub-module203, the master plan builder sub-module 205, and the inventoryestimation sub-module 207) can be coupled to the one or more data stores210,220,230,250,260,270,280 and can read from and write to the one ormore data stores 210,220,230,250,260,270,280.

Data store 210 can store recommendations 213 and document parameters215. The recommendations 213 can be provided by content providers. Arecommendation 213 is a representation of digital content. Arecommendation 213 can include, for example, and is not limited to, oneor more text, links, thumbnails, images, icons, etc.

The document parameters 215 can describe parameters for publisherdocuments (e.g., documents 150,160 in FIG. 1). The document parameters215 can be different for the documents of the various publishers and/orfor the various documents. The document parameters 215 can describe, forexample, the number of recommendation opportunities (e.g.,recommendation opportunities 155,165 in FIG. 1) for each destinationdocument. The document parameters 215 describe which the types ofrecommendations 213 that are eligible to be served as impressions topopulate the recommendation opportunities in the documents. Otherexamples of document parameters 215 can include, for example, and arenot limited to language, country, device platform, application operatingsystem platform, etc. For example, the document parameters 215 forDocument-XYZ may indicate that recommendations 213 for the mobile deviceplatform and the English language may populate the recommendationopportunities in Document-XYZ.

Data store 220 can store campaign data 221. The campaign data 221 can beprovided by multiple content providers. The campaign data 221 caninclude parameters, such as, and not limited to, identifiers of thecontent for use by the campaign, a total budget (B) for the campaign,the maximum cost per click (CPC) for the campaign, the desired averageCPC for the campaign, and links to the recommendations for use by thecampaign, etc. For example, the data store 210 may store one hundredrecommendations (e.g., Recommendation-1 to Recommendation-100) forContent-Provider-A. The campaign data 221 for Content-Provider-A mayspecify twenty recommendation identifiers out of the one hundredrecommendations that should be used by Campaign-A, links to the twentyrecommendations, a total budget B of $1,000 for Campaign-A, a maximumCPC of $1 for Campaign-A and a desired average CPC of 70 cents forCampaign A.

The inventory estimation sub-module 207 can determine the expectednumber of views for the various destination documents (e.g., documents150,160 in FIG. 1). For example, the inventory estimation sub-module 207may predict that Document-XYZ may have 1 million views per day. Inanother example, the inventory estimation sub-module 207 may predictthat Document-DEF may have 10 million views per day. The inventoryestimation sub-module 207 can use the user activity data 281 stored in adata store 280 that is coupled to the optimization module 200, one ormore prediction models, statistical analysis, etc. to determine theexpected number of views for a document. The prediction models can bestored in data store 280. The user activity data 281 can includehistorical data that can be used to determine the expected number ofviews for a destination document. The inventory estimation sub-module207 can store the expected number of views (e.g., predicted views 261)for the various documents in the data store 260.

In one implementation, the inventory estimation sub-module 207 canpredict the number of views for a destination document based on data forthe subsections of a publisher network. The inventory estimationsub-module 207 can identify a subsection of the publisher network usingpublisher subsection data 283. The publisher subsection data 283 cangroup documents of the various publishers based on one or moreattributes, such as, and not limited to, category and sub-category. Acategory can include one or more sub-categories. An example of acategory can include, and is not limited to news, and an example of asub-category for news can include, and is not limited to, internationalnews. The publisher subsection data 283 can assign the documents of thevarious publishers to one or more categories and/or sub-categories. Theassignments can be based on user (e.g., system administrator, contentprovider administrator) input. In one implementation, the inventoryestimation sub-module 207 can predict the number of views for adestination document based on an average number of views of documents inthe same category and/or sub-category as the destination document. Forexample, Document-XYZ may be in the news category and the sports newssub-category. The average number of views for the documents in thesubsection of sports news in the publisher network may be 500,0000 viewsper day, and the inventory estimation sub-module 207 may predict thatDocument-XYZ will have 500,0000 views per day.

The optimization module 200 may collect registration information,personal information, and/or demographic information pertaining to usersand/or groups of users (e.g., age of user groups, geography (e.g.,country, state, city) of user groups) and may store the information inone or more data stores (e.g., hard disks, memories, databases, etc.)(e.g., data store 210,280). In situations in which the implementationsdiscussed herein collect personal information and/or demographic aboutusers, or may make use of personal information and/or demographicinformation, the user may be provided with an opportunity to controlwhether the optimization module 200 is allowed collect the information(e.g., information about a user's social network, social actions oractivities, profession, a user's preferences, or a user's currentlocation), or to control whether and/or how to receive content from thecontent sharing platform that may be more relevant to the user. Inaddition, certain data may be treated in one or more ways before it isstored or used, so that personally identifiable information is removed.Thus, the user may have control over how, when, and what type ofinformation is collected about the user and used by the optimizationmodule 200.

The master plan builder sub-module 205 can generate an allocation plan551. The master plan builder sub-module 205 can use the number ofpredicted views 261, the number of recommendation opportunities (e.g.,recommendation opportunities 155 in FIG. 1) for each destinationdocument (e.g., Document-XYZ 150 in FIG. 1) as specified in the documentparameters 215, other document parameters (e.g., language, country,device platform, application operating system platform, etc.) specifiedin the document parameters 215, and the campaign data 221 to generate anallocation plan 251. The master plan builder sub-module 205 can also usethe historical data accumulated in the data stores 230 and 240. Themaster plan builder sub-module 205 can store the allocation plan 251 inthe data store 250. The allocation plan 251 is a plan for the aggregateof the campaign data 221 from the content providers. The allocation plan251 can indicate the desired number of impressions for therecommendations 213 for the various destination documents. An impressionis an instance of populating a destination document (e.g., webpage,mobile application GUI) (e.g., Document-XYZ in FIG. 1) with a contentrecommendation 213 (e.g., recommendations 107,109,11 in FIG. 1). Theallocation plan 251 can include quotas at the recommendation level. Forexample, Content-Provider-A may have Recommendation-1 (e.g., onlinearticle) as part of Campaign-A. The allocation plan 251 may indicatethat “15,000 impressions” on Document-XYZ should be allocated toRecommendation-1 based on the document parameters 215 (e.g., language,country, device platform, application operating system platform, numberof recommendation opportunities, etc.) for Document-XYZ. Implementationsof creating an allocation plan are described in greater detail below inconjunction with FIG. 4 and FIG. 5.

The user selection probability estimation sub-module 201 can determineselection probabilities 231 for recommendations 213 and can store theselection probabilities 231 in the data store 230. The selectionprobability 231 is the likelihood that a user (e.g., user 161A inFIG. 1) may select an impression of a recommendation 213 that populatesa recommendation opportunity (e.g., opportunity 155 in FIG. 1) in adestination document (e.g., Document-XYZ 150 in FIG. 1) within thecontext, for example, of the user's current session with the destinationdocument. For example, User-Jane-Doe may view destination Document-XYZto read a new book review. The user selection probability estimationsub-module 201 can estimate how likely User-Jane-Doe will select arecommendation 213 if the recommendation 213 is served as an impressionon Document-XYZ within the context of User-Jane-Doe reading the bookreview. For example, the user selection probability estimationsub-module 201 may determine that Recommendation-1 has a selectionprobability 231 of 5%, Recommendation-2 has a selection probability 231of 2%, Recommendation-3 has a selection probability 231 of 60%, etc.

The user selection probability estimation sub-module 201 can use theuser parameters 217 for the particular user (e.g., User-Jane-Doe) storedin the data store 210, prediction models, and statistical analysis todetermine the selection probability 231. The user parameters 217 caninclude, for example, and are not limited to, location information ofthe user, device information of the user, user preferences, userinterests, etc. The user selection probability estimation sub-module 201can also determine the selection probabilities 231 for impressions ofrecommendations that have not yet been served to destination documents.

The real-time ranking, allocation and pricing sub-module 203 canallocate the recommendations 213 to the destination documents based on alottery. In one implementation, the real-time ranking, allocation andpricing sub-module 203 allocates the recommendations 217 to thedestinations documents based on a lottery using the allocation plan 251and the selection probabilities 231.

The real-time ranking, allocation and pricing sub-module 203 can includea ranking component 291, an allocation component 293, and a pricingcomponent 295. The ranking component 291 can determine the grades 241for the recommendations 213 in the allocation plan 251 and store thegrades 241 in the data store 240. In one implementation, the rankingcomponent 291 determines the grades 241 based on the selectionprobabilities 231 and the maximum CPC a buyer (e.g., content provider)is willing to pay. For example, the grade 241 for a recommendation 213in the allocation plan 251 may be calculated as the product of theselection probability 231 for the recommendation 213 multiplied by theCPC for the recommendation 213. For example, Recommendation-1 may beassigned a grade 241 of “5” and Recommendation-2 may be assigned a grade241 of ‘7’. In another example, the grades 241 may be calculated as theproduct of (selection probability)^(α)×(CPC)^(β). In one implementation,α and β are values selected by the ranking component 291, for example,based on configuration data that may be stored in a data store (e.g.,data store 240). In one implementation, the optimization module 200 canbe coupled to another system that determines the grades 241, which thereal-time ranking, allocation and pricing sub-module 203 can access. Theranking component 291 can use alternative computations to determine thegrades 241.

The allocation component 293 can use the grades 241 calculated by theranking component 291 to derive the lottery probabilities 243 for therecommendations 213 in the allocation plan 251 based on the proportionsof the grades 241. The allocation component 293 can process any set ofgrades 241 and is not limited to the grade calculations described above.The lottery probability 243 is the probability that a recommendation 213in the allocation plan 251 may be selected for allocation to be servedas an impression in a destination document. The lottery probabilities243 can be stored in the data store 240. In one implementation, theallocation component 293 calculates the lottery probability 243 for eachof the recommendations 213 in the allocation plan 251 as a proportion ofthe grade 241 of the particular recommendation 213 to the total of thegrades 241 of the recommendations 213.

For brevity and simplicity, an example illustrating the calculation ofthe lottery probabilities 243 is described using two recommendations 213in an allocation plan 251. For example, Recommendation-1 may be assigneda grade 241 of “5” and Recommendation-2 may be assigned a grade 241 of‘7’. The total of the grades 241 is “12”. The allocation component 293may determine that Recommendation-1 has a lottery probability 243 of5/12, and that Recommendation-2 has a lottery probability 243 of 7/12.

The allocation component 293 can determine the allocations 245 of therecommendations 213 based on the proportion of the grades (i.e., thelottery probabilities 251). The allocations 245 can be stored as anallocation order in the data store 240. For example, the allocationcomponent 293 may select the recommendations 213 for allocation in orderof lottery probability 243. For example, the recommendation 213 havingthe highest lottery probability 243 may be selected first, etc. Theallocation component 293 can position the first selected recommendation213 in the most favorable position (e.g., top) in the allocation order(e.g., allocations 245). The allocation component 93 can use theallocation plan 251, together with the grades 241, to determine thelottery probabilities 243. Recommendations 213 with high allocation planpriority can be assigned a higher lottery probability 243 than the onewarranted by its respective grade 241. In one implementation, theallocation of recommendations 213 to positions on the document isperformed by sequential lottery. For example, there may be two positionsfor recommendations, and Recommendation-A may have a lottery probabilityof 0.7, Recommendation-B may have a lottery probability of 0.2, andRecommendation-C may have a lottery probability of 0.1. The allocationcomponent 293 can perform the lottery for position one, with the abovelottery probabilities. For example, the lottery can selectRecommendation A for position one. Then, the allocation component 293can perform a second lottery for position two, which may selectRecommendation-C (even though Recommendation-B has a higher lotteryprobability).

The pricing component 295 can determine the price (e.g., CPC) a buyer(e.g., content provider) should pay for a user selection of arecommendation 213 in the allocations 245. A buyer may specify a maximumCPC for a campaign in the campaign data 221. The price that isdetermined by the pricing component 295 may be less than or equal to themaximum CPC. The pricing component 295 can store the optimal pricing 247in the data store 240. The pricing component 295 can also take intoaccount the allocated position for the recommendation.

The allocation component 293 can allocate each of the recommendations213 in the allocations 245 to a specific position to create a rankedrecommendation list 271 and can store the ranked recommendation list 271in the data store 270. The real-time ranking, allocation and pricingsub-module 203 can provide the ranked recommendation list 271 to asystem (e.g., serving system) to serve the recommendations 213 to theservers hosting the destination documents, for example, based on theranking order in the ranked recommendation list 271. For eachrecommendation 213 in the allocations 245, for example, the rankedrecommendation list 271 can include, and is not limited to, arecommendation identifier, a rank identifier, a price, the associatedcontent provider, etc.

FIG. 3 is a flow diagram of an implementation of a method 300 forranking, allocating and pricing recommendations. The method may beperformed by processing logic that may comprise hardware (circuitry,dedicated logic, etc.), software (such as is run on a general purposecomputer system or a dedicated machine), or a combination of both. Inone implementation, the method 300 may be performed by the optimizationmodule 145 hosted in a server 140 of FIG. 1 or by the optimizationmodule 200 hosted in FIG. 2.

At block 301, the server receives campaign data from one or more contentproviders. For example, the server receives campaign data for a campaignfrom Content-Provider-A and campaign data for a campaign fromContent-Provider-B. The campaign data can include parameters, such as,and are not limited to, identifiers of the recommendations for use bythe campaign, a total budget (B) for the campaign, the maximum CPC forthe campaign, the desired average CPC for the campaign, links to therecommendations for use by the campaign, etc. For example, the servermay store fifty recommendations provided by Content-Provider-A, and thecampaign data for Content-Provider-A may specify that twenty of thoserecommendations (e.g., online articles, web blogs, social network posts,data feeds, etc.) should be used in the campaign for Content-Provider-A.

At block 303, the server creates an allocation plan based on thecampaign data from the content providers and the predicted number ofviews for the destination documents. The allocation plan can describethe optimal number of impressions for the recommendations that will beserved on the various destination documents. For example,Content-Provider-A may have Recommendation-1 (e.g., online article) aspart of a campaign. The allocation plan may indicate that “15,000impressions” on Document-XYZ should be allocated to Recommendation-1.Implementations of creating an allocation plan are described in greaterdetail below in conjunction with FIG. 4 and FIG. 5.

At block 305, the server determines selection probabilities based onuser activity data. The selection probabilities represent the likelihoodthat a user will select a recommendation that is served as an impressionin the destination document based on the context of the user's currentsession with the destination document. The server can determine theselection probabilities in real time. The allocation plan may indicatethat “5,000 impressions” on Document-XYZ should be allocated toRecommendation-1 and “10,000 impressions” on Document-XYZ should beallocated to Recommendation-2. The server can estimate how likelyUser-Jane-Doe will select each of the recommendations (e.g.,Recommendation-1, Recommendation-2) in the allocation plan. For example,the server may determine that Recommendation-1 has a selectionprobability 231 of 5% and that Recommendation-2 has a selectionprobability 231 of 2%.

At block 307, the server calculates the grades for each of therecommendations in the allocation plan. The calculated grades induce anatural ranking of the recommendations by their grade value. The servercan use the selection probabilities and additional recommendationparameters such as the maximum CPC a buyer (e.g., content provider) iswilling to pay, to determine grades for the recommendation in theallocation. In one implementation, the grade is calculated as theproduct of the selection probability for the recommendation multipliedby the maximum or desired CPC for the recommendation.

At block 309, the server allocates the recommendations to thedestination document and positions based on the allocation plan, gradesand selection probabilities. The server can use the grades to determinelottery probabilities for the recommendations in the allocation plan.The lottery probability is the probability that a recommendation in theallocation plan may be selected for allocation to be served as animpression in a destination document. In one implementation, the servercalculates the lottery probability for each of the recommendations inthe allocation plan as a proportion of the grade of the particularrecommendation to the total of the grades of the recommendations. In thepreferred implementation, the server adjusts the proportion-based grade.

The server can use the lottery probabilities to allocate therecommendations to the destination documents. For example, the servermay select the recommendations for allocation in order of lotteryprobability. For example, the server may allocate the recommendationhaving the highest lottery probability first and may position therecommendation that is allocated first in the most favorably position(e.g., top) in the allocation order.

At block 311, the server determines pricing for the recommendationsbased on the allocations (e.g., allocation order), recommendationsparameters and allocation plan. One implementation of a grade-adjustedallocation and pricing method that encourages agents to use their truevalue as their maximum CPC bid is described below in conjunction withFIG. 6.

FIG. 4 is a flow diagram of an implementation of a method 400 forcreating an allocation plan. The method may be performed by processinglogic that may comprise hardware (circuitry, dedicated logic, etc.),software (such as is run on a general purpose computer system or adedicated machine), or a combination of both. In one implementation, themethod 400 may be performed by the optimization module 145 hosted in aserver 140 of FIG. 1 or by the optimization module 200 hosted in FIG. 2.

At block 401, the server determines the expected number of views for thedestination documents. For example, the server may predict thatDocument-XYZ may have 5,000 views per day. In one implementation, theserver uses user activity data, one or more prediction models,statistical analysis, etc. to determine the expected number of views forthe destination document. In another example, the server may determinethat Document-XYZ may have 1 million views per day. In anotherimplementation, the server predicts the number of views for adestination document based on data for the subsections of a publishernetwork. For example, Document-XYZ may be in the news category and thesports news sub-category, and the data for the subsections of thepublisher network may indicate there is an average number of views forthe documents in the sports news subsection of 8,0000 views per day. Theserver may use the average of 8,0000 views per day as the predictednumber of views per day for Document-XYZ.

At block 403, the server identifies the number of recommendationopportunities per view for the destination documents. The publishers mayspecify the number of recommendation opportunities (e.g., recommendationopportunities 155,165 in FIG. 1) per view for the destination documents.For example, the publisher GlobalNews, Inc. may specify thatDocument-XYZ (e.g., GlobalNews webpage) pertains to Subsection-Y (e.g.,sub-category international news) and has three recommendationopportunities per view. In another example, the publisherEntertainmentNews, Inc. may specify that Document-DEF (e.g.,EntertainmentNews webpage) pertains to Subsection-W (e.g., sub-categoryentertainment news) and has five recommendation opportunities per view.

At block 405, the server determines the total number of recommendationopportunities for each destination document. For example, the server maydetermine that the total number of recommendation opportunities forDocument-XYZ for Subsection-Y is 15,000 recommendation opportunities,which is the product of 5,000 expected views per day multiplied by threerecommendation opportunities per view. In another example, the servermay determine that the total number of recommendation opportunities forDocument-DEF for Subsection-W is 40,000 recommendation opportunities,which is the product of 8,000 expected views per day multiplied by fiverecommendation opportunities per view.

At block 407, the server identifies the campaign parameters associatedwith each recommendation. For example, Recommendation-1 may be fromContent-Provider-A for Campaign-A. Content-Provider-A may have specifiedthat Campaign-A has parameters that include a budget (B) of $1000 and acost-per-click (CPC) of $1. In another example, Recommendation-2 may befrom Content-Provider-B for Campaign-B. Content-Provider-B may havespecified that Campaign-B has parameters that include a budget of $800and a cost-per-click (CPC) of $2.

At block 409, the server determines a predicted CTR for the combinationsof recommendations and destination documents. The predicted CTRrepresents the predicted rate that the recommendation will be selectedat when served as an impression. The predicted CTR can be determinedfrom user activity data that is stored in the data store. For example,the server may predict that upon Document-XYZ displayingRecommendation-1 from Content-Provider-A the combination may result in apredicted CTR of 4%.

At block 411, the server determines an optimal allocation of clicks andimpressions for the recommendations for the combinations subject to thecampaign parameters. A click hereinafter refers to a selection of animpression in a document. The selection may be made via an input device(e.g., touch screen, mouse, keyboard, motion sensor, remote control,voice, etc.). The server can use the budget B, CPC, CTR, and totalnumber of recommendation opportunities for each combination (e.g.,recommendation/document combination) to determine the optimal allocationof clicks. For example, Content-Provider-A may have agreed to pay thethird party service provider a maximum budget B of $1000 and a maximumof $1 CPC. In another example, Content-Provider-B may have agreed to paythe third party service provider a maximum budget B of $800 and amaximum of $2 CPC.

The server can use linear programming to determine the optimalallocation of clicks and impressions for the recommendations based onthe maximum budgets and maximum cost-per-clicks (CPCs). For example, leti be a destination document (e.g., Document-XYZ, Document-DEF) and j bea set of campaign parameters, which can include the budget (B_(j)) forthe campaign and the cost per click (CPC_(j)) for the campaign. Theserver can use the click through rate (CTR_(ij)), campaign parameters,and total number of recommendation opportunities for a destinationdocument i to determine, N_(ij), the total number of impressions for arecommendation on destination document i, such that N_(ij) does notexceed the budget B_(j) and the CPC_(j).Σ_(ij) N _(ij) CPC _(j) CTR _(ij) ≤B _(j)  (Constraint C1)Σ_(ij) N _(ij) ≤N _(j)  (Constraint C2: inventory estimate)max(Σ_(ij) N _(ij) CPC _(j) CTR _(ij)) Subject to constraints  (Objective Function O1)

For example, the server may use linear programming optimizes O1 subjectto C1 and C2 to determine the optimal allocations of clicks andimpressions (N_(ij)) for each recommendation. The allocation plan can bethe optimal number of impressions (N_(ij)) for each recommendation.

FIG. 5 illustrates an example of creating an allocation plan 550,according to various implementations. The allocation plan 550 canspecify the optimal number of impressions allocated to a recommendation(e.g., Recommendation-1, Recommendation, etc.). There can be multiplerecommendations. For simplicity and brevity two recommendations 511,513are shown. Portion 560 of FIG. 5 represents the inputs of theoptimization module (e.g., optimization module 145 in FIG. 1,optimization module 200 in FIG. 2). Another portion 570 represents anintermediate solution in terms of clicks. Another portion 580 representsa translation of the intermediate solution to impression terms.

The server may use the campaign parameters (e.g., budgets, CPC, etc.)for the various campaigns to create the allocation plan 550. Forexample, Content-Provider-A may specify parameters 509 for a campaignthat includes a budget of $1000 and a CPC of $1. In another example,Content-Provider-B may specify parameters 509 for a campaign thatincludes a budget of $800 and a CPC of $2. The server can associate therespective parameters with the various recommendations 511,515 from thecontent providers.

The server may use the total number of recommendation opportunities forthe various destination documents to create the allocation plan 550. Thetotal number of recommendation opportunities can be determined as theproduct of the number of expected views that is predicted for documentsin a subsection of the publisher network multiplied by the number ofrecommendation opportunities on a destination document per view. Forexample, the server may determine that destination Document-XYZ 503 forSubsection-Y has 15,000 total recommendation opportunities 501. Forexample, the server may predict that Subsection-Y has 5,000 expectedviews and that there are three recommendation opportunities per view forDocument-XYZ 503. In another example, the server may determine thatdestination Document-DEF 505 for Subsection-W has 40,000 totalrecommendation opportunities 507. For example, the server may predictthat Subsection-W has 8,000 expected views and that there are fiverecommendation opportunities per view for Document-DEF 505.

The server can estimate the click-through-rates (CTRs) 520 for thevarious combinations of recommendations 511,513 and destinationdocuments 503,505. For example, the server may estimate that uponDocument-XYZ 503 displaying Recommendation-1 511 from Content-Provider-Athe document and recommendation combination may result in a CTR of 4%.In another example, the server may estimate that upon Document-XYZ 503displaying Recommendation-2 513 from Content-Provider-B thedocument/recommendation combination may result in a CTR of 2%. Inanother example, the server may estimate that upon Document-DEF 505displaying Recommendation-1 511 from Content-Provider-A thedocument/recommendation combination may result in a CTR of 0.5%. Inanother example, the server may estimate that upon Document-DEF 505displaying Recommendation-2 513 from Content-Provider-B the combinationmay result in a CTR of 2%.

The server can determine an optimal number of clicks 530 and an optimalnumber of impressions 550 for the recommendations 511,513 for thevarious document/recommendation combinations. The allocation plan 550 isthe optimal number of impressions 550 for the recommendations 511,513for the various document/recommendation combinations. The allocationplan 550 can include quotas (e.g., “15,000 impressions” 551, “0impressions” 553, “20,000 impressions” 555, “20,000 impressions” 557) atthe recommendation level. The server can use linear programming, and forexample, Objection Function 1 subject to Constraint 1 and Constraint 2above, to determine the optimal number of clicks 530 and the optimalnumber of impressions 550 while taking into account the budgets, CPCs,CTRs, and total number of recommendation opportunities.

For example, the server may determine that the optimal number of clicks530 includes “600 clicks” to Recommendation-1 511 for Document-XYZ 503,“100 clicks” to Recommendation-1 511 for Document-DEF 505, “0 clicks” toRecommendation-2 513 for Document-XYZ 503, and “400 clicks” toRecommendation-2 513 for Document-DEF 505. For example, the server maydetermine that if all of the 15,000 recommendation opportunities 501 areallocated as “15,000 impressions” 551 to Recommendation-1 511, and thereis a CTR of 4%, the number of clicks is (15,000)*(0.04)=“600 clicks”531. If all of the 15,000 recommendation opportunities 501 are allocatedto Recommendation-1 511, then “0 impressions” 553 for Document-XYZ 503would be allocated to Recommendation-2 515, resulting in “0 clicks” 533for Recommendation-2 513 for Document-XYZ 503. If Content-Provider-B hasa budget B of $800 and a CPC of $2, then the number of clicks is$800/$2=“400 clicks” 537, which may all be allocated to Recommendation-2513 for Document-DEF 505.

If “400 clicks” 537 are allocated to Recommendation-2 513 forDocument-DEF 505, the number of impressions allocated toRecommendation-2 513 for Document-DEF 505 is the number of clicksdivided by the CTR, which is (400 clicks)/(0.02)=“20,000 impressions”557. If Document-DEF 505 has a total number of recommendationopportunities of “40,000 recommendation opportunities” 507 and “20,000impressions” 557 are allocated to Recommendation-2 513, then theremaining “20,000 impressions” 555 can be allocated to Recommendation-1511.

The allocation of the “20,000 impressions” 555 for Document-DEF 505 toRecommendation-1 511 can result in “100 clicks” 535. If the CPC is $1and there are “100 clicks” 535, then Content-Provider-A should pay $100(e.g., (20,000)*(0.5%)*($1)=$100) to the third party service provider.With the allocation plan 550, Content-Provider-A is predicted to pay$100 for the 20,000 impression quota for Document-DEF 505 and $600(e.g., (15,000)*(4%)*($1)=$600) for the 15.000 impression quota forDocument-XYZ 503, which is a total of $700 and less than the overallbudget B of $1000 for Content-Provider-A. With the allocation plan 550,Content-Provider-B is predicted to pay $0 for the “0 impressions” 553quota for Document-XYZ 503 and $800 (e.g., (20,000)*(2%)*($2)=$600) forthe 20,000 impression quota for Document-DEF 505, which is a total of$800 and equal to the overall budget B of $800 for Content-Provider-B.

FIG. 6 is a flow diagram of one implementation of a method 600 fordetermining the allocation and pricing for recommendations. The methodmay be performed by processing logic that may comprise hardware(circuitry, dedicated logic, etc.), software (such as is run on ageneral purpose computer system or a dedicated machine), or acombination of both. In one implementation, the method 600 may beperformed by the optimization module 145 hosted in a server 140 of FIG.1 or by the optimization module 200 hosted in FIG. 2, and especiallymodule 203 of FIG. 2.

The method 600 implements a grade-adjusted allocation and pricingmechanism that encourages agents to use their true value as theirmaximum CPC (Max-CPC) bid. Method 600 can encourage buyers (e.g.,content providers) to specify a maximum CPC in the campaign parametersthat is close to the buyer's actual value of a click. The server canadjust the grades that are assigned to the recommendations and candetermine prices for the recommendations according to the actual demandreflected through the allocation list (e.g., allocation list generatedin block 307 in FIG. 3).

At block 601, the server identifies a grade for each recommendation andderives from it a lottery probability (e.g., assigned in block 309 inFIG. 3) for a recommendation. In one implementation, the servercalculates the lottery probability (p) for the recommendation as aproportion of the grade of the particular recommendation to the total ofthe grades of the recommendations. For example, the grade (p_(i)) forRecommendation-1 may be

$p_{1} = {\frac{7}{12} = {0.58.}}$At block 603, the server identifies the number of impressions (N_(i))assigned to the recommendation for a particular destination documentfrom an allocation plan. For example, the number of impressions (N₁) forRecommendation-1 for Document-DEF may be N₁=20,000.

At block 605, the server determines whether the expected number ofimpressions for the recommendation is more than requested by the plan,in which case the server would need to decrease the original lotteryprobability so the expected number of impressions matches the number ofimpressions (N_(i)) from the allocation plan. Let N be the total numberof recommendation opportunities for the destination document. Forexample, for Document-DEF, N=40,000. The server can use Statement 1 andStatement 2 to determine whether to decrease the original grade.If (p _(i) ×N)≥N _(i), then determine decreased lottery probability p′_(i)  (Statement 1)If (p ₁ ×N)<N _(i), then do not decrease lottery probability p_(i)  (Statement 2)

If the server should decrease the original lottery probability, theserver determines the decreased lottery probability at block 607 andassigns the decreased lottery probability to the recommendation at block609. For example, for Recommendation-1 for Document-DEF, let:

$\begin{matrix}{{N = {40\text{,}000}},{N_{1} = {20\text{,}000}},{{{and}\mspace{14mu} p_{1}} = {\frac{7}{12} = 0.58}}} & \left( {{Statement}\mspace{14mu} 3} \right) \\{{p_{1} \times N} = {{0.58 \times 40\text{,}000} = {23\text{,}200}}} & \left( {{Statement}\mspace{14mu} 4} \right) \\{p_{1}^{\prime} = {\frac{N_{1}}{N} = {\frac{20\text{,}000}{40\text{,}000} = 0.50}}} & \left( {{Statement}\mspace{14mu} 5} \right)\end{matrix}$

The server may determine for Recommendation-1 for Document-DEF thatp₁×N≥N₁ (e.g., 0.58≥0.50) and can determine the decreased lotteryprobability at block 607. In one implementation, the server determinesthe decreased lottery probability to be:

$\begin{matrix}{\frac{N_{1}}{N} = {{decreased}\mspace{14mu}{lottery}\mspace{14mu}{probability}\mspace{14mu} p_{1}^{\prime}}} & \left( {{Statement}\mspace{14mu} 6} \right)\end{matrix}$

For example, the server may determine the decreased lottery probability,p′₁, for Recommendation-1 for Document-DEF is

$\frac{N_{i}}{N} = {\frac{20\text{,}000}{40\text{,}000} = 0.50}$and can assign the decreased lottery probability as p_(i) toRecommendation-1 for Document-DEF at block 609.

At block 611, the server determines the prices for the recommendationbased on the lottery probability. In one implementation, the serverdetermines the price for Recommendation-i using Equation 1:

$\begin{matrix}{{{Price}(i)} = {{MaxCPC}_{i} \times \left( \frac{N_{i}}{N \times p_{i}} \right)\mspace{14mu}{if}\mspace{14mu}{the}\mspace{14mu}{lottery}\mspace{14mu}{probability}\mspace{14mu}{was}\mspace{14mu}{decreased}\mspace{14mu}\left( {{see}\mspace{14mu}{{stmt}.\mspace{14mu} 1}} \right)}} & \left( {{Eq}.\mspace{14mu} 1} \right)\end{matrix}$

Price(i)=MaxCPC_(i) if the lottery probability was not decreased (seestmt. 2)

For example, for Recommendation-1 for Document-DEF, discussed above withp1=0.58 and p1′=0.5; let MaxCPC₁=$1. The server may determine that theprice for Recommendation-1 is:

$\begin{matrix}{{{Price}(1)} = {{{\$ 1} \times \left( \frac{20\text{,}000}{40\text{,}000 \times 0.58} \right)} = {{\$ 0}{.86}}}} & \left( {{Statement}\mspace{14mu} 7} \right)\end{matrix}$

FIG. 7 is a block diagram of an example computer system 700 that mayperform one or more of the operations described herein, in accordancewith various implementations In alternative implementations, the machinemay be connected (e.g., networked) to other machines in a LAN, anintranet, an extranet, or the Internet. The machine may operate in thecapacity of a server or a client machine in client-server networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine may be a personal computer (PC), atablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), acellular telephone, a web appliance, a server, a network router, switchor bridge, or any machine capable of executing a set of instructions(sequential or otherwise) that specify actions to be taken by thatmachine. Further, while only a single machine is illustrated, the term“machine” shall also be taken to include any collection of machines thatindividually or jointly execute a set (or multiple sets) of instructionsto perform any one or more of the methodologies discussed herein.

The example computer system 700 includes a processing device (processor)702, a main memory 704 (e.g., read-only memory (ROM), flash memory,dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM),double data rate (DDR SDRAM), or DRAM (RDRAM), etc.), a static memory706 (e.g., flash memory, static random access memory (SRAM), etc.), anda data storage device 714, which communicate with each other via a bus730.

Processor 702 represents one or more general-purpose processing devicessuch as a microprocessor, central processing unit, or the like. Moreparticularly, the processor 702 may be a complex instruction setcomputing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction word (VLIW) microprocessor,or a processor implementing other instruction sets or processorsimplementing a combination of instruction sets. The processor 702 mayalso be one or more special-purpose processing devices such as anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), network processor,or the like. The processor 702 is configured to execute instructions 722for performing the operations and steps discussed herein.

The computer system 700 may further include a network interface device704. The computer system 700 also may include a video display unit 710(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), analphanumeric input device 712 (e.g., a keyboard), a cursor controldevice 714 (e.g., a mouse), and a signal generation device 716 (e.g., aspeaker).

The data storage device 714 may include a computer-readable storagemedium 724 on which is stored one or more sets of instructions 722(e.g., software) embodying any one or more of the methodologies orfunctions described herein. The instructions 722 may also reside,completely or at least partially, within the main memory 704 and/orwithin the processor 702 during execution thereof by the computer system700, the main memory 704 and the processor 702 also constitutingcomputer-readable storage media. The instructions 722 may further betransmitted or received over a network 720 via the network interfacedevice 708.

In one implementation, the instructions 722 include instructions for anoptimization module (e.g., optimization module 145 in FIG. 1) and/or asoftware library containing methods that call the optimization module.While the computer-readable storage medium 728 (machine-readable storagemedium) is shown in an exemplary implementation to be a single medium,the term “computer-readable storage medium” should be taken to include asingle medium or multiple media (e.g., a centralized or distributeddatabase, and/or associated caches and servers) that store the one ormore sets of instructions. The term “computer-readable storage medium”shall also be taken to include any medium that is capable of storing,encoding or carrying a set of instructions for execution by the machineand that cause the machine to perform any one or more of themethodologies of the present disclosure. The term “computer-readablestorage medium” shall accordingly be taken to include, but not belimited to, solid-state memories, optical media, and magnetic media.

In the foregoing description, numerous details are set forth. It will beapparent, however, to one of ordinary skill in the art having thebenefit of this disclosure, that the present disclosure may be practicedwithout these specific details. In some instances, well-known structuresand devices are shown in block diagram form, rather than in detail, inorder to avoid obscuring the present disclosure.

Some portions of the detailed description have been presented in termsof algorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, for reasons of common usage, to refer tothese signals as bits, values, elements, symbols, characters, terms,numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “receiving”, “determining”, “ranking”, “allocating”,“providing”, “assigning”, “creating”, “selecting”, “pricing”, or thelike, refer to the actions and processes of a computer system, orsimilar electronic computing device, that manipulates and transformsdata represented as physical (e.g., electronic) quantities within thecomputer system's registers and memories into other data similarlyrepresented as physical quantities within the computer system memoriesor registers or other such information storage, transmission or displaydevices.

For simplicity of explanation, the methods are depicted and describedherein as a series of acts. However, acts in accordance with thisdisclosure can occur in various orders and/or concurrently, and withother acts not presented and described herein. Furthermore, not allillustrated acts may be required to implement the methods in accordancewith the disclosed subject matter. In addition, those skilled in the artwill understand and appreciate that the methods could alternatively berepresented as a series of interrelated states via a state diagram orevents. Additionally, it should be appreciated that the methodsdisclosed in this specification are capable of being stored on anarticle of manufacture to facilitate transporting and transferring suchmethods to computing devices. The term article of manufacture, as usedherein, is intended to encompass a computer program accessible from anycomputer-readable device or storage media.

Certain implementations of the present disclosure also relate to anapparatus for performing the operations herein. This apparatus may beconstructed for the intended purposes, or it may comprise ageneral-purpose computer selectively activated or reconfigured by acomputer program stored in the computer. Such a computer program may bestored in a computer readable storage medium, such as, but not limitedto, any type of disk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions.

Reference throughout this specification to “one implementation” or “animplementation” means that a particular feature, structure, orcharacteristic described in connection with the implementation isincluded in at least one implementation. Thus, the appearances of thephrase “in one implementation” or “in an implementation” in variousplaces throughout this specification are not necessarily all referringto the same implementation. In addition, the term “or” is intended tomean an inclusive “or” rather than an exclusive “or.” Moreover, thewords “example” or “exemplary” are used herein to mean serving as anexample, instance, or illustration. Any aspect or design describedherein as “exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects or designs. Rather, use of the words“example” or “exemplary” is intended to present concepts in a concretefashion.

It is to be understood that the above description is intended to beillustrative, and not restrictive. Many other implementations will beapparent to those of skill in the art upon reading and understanding theabove description. The scope of the disclosure should, therefore, bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

What is claimed is:
 1. A method comprising: receiving campaign data froma first content provider, wherein the campaign data comprises a set ofcontent links, a target cost per click for the set of content links, anda budget amount selected by the first content provider; creating anallocation plan comprising assignments of the set of content links fordisplay in a plurality of documents, wherein the allocation plancomprises a target number of impressions for the set of content links;collecting user activity data associated with a first user; determininga first user-specific selection probability corresponding to a firstcontent link of the set of content links based on the user activity dataassociated with the first user, wherein the first user-specificselection probability represents a calculated probability of servingcontent in response to an interaction by the first user with the firstcontent link; assigning, to the first content link, a first graderepresenting a product of the first user-specific selection probabilityassociated with the first user and the first content link and the targetcost per click selected by the first content provider; determiningpricing for provisioning the first content link to one or more of theplurality of documents; generating a plurality of sub-categories toclassify the plurality of documents; determining, using a firstprediction model, a predicted number of views for a first document ofthe plurality of documents, wherein the predicted number of views isbased on an average number of views of a first set of documents in afirst sub-category comprising the first document; allocating, by aprocessing device, the first content link to the first document based onthe first grade, the allocation plan, the pricing, the predicted numberof views, and the first user-specific selection probability; and inresponse to the first user requesting the first document, causing, basedon the allocating, the first content link to be served via a designatedportion of the first document delivered to the first user.
 2. The methodof claim 1, further comprising: generating a ranking of a plurality ofrecommendations based on a plurality of grades; and providing theranking to a distribution system to serve the plurality ofrecommendations to one or more servers providing the plurality ofdocuments.
 3. The method of claim 2, wherein the allocating comprises:determining a first lottery probability of the first content link beingselected in a lottery based on the first grade, wherein the allocatingis further based on the first lottery probability.
 4. The method ofclaim 3, wherein determining the first lottery probability comprises:determining a proportion of the first grade to a total of the pluralityof grades.
 5. The method of claim 3, wherein determining pricing forprovisioning the first content link comprises: determining a decrease tothe first lottery probability for the first content link; anddetermining an adjusted price for provisioning of the first content linkbased on the decrease to the first lottery probability.
 6. A systemcomprising: a memory to store instructions; and a processing deviceoperatively coupled with the memory, the processing device to executethe instructions to: receive campaign data from a first contentprovider, wherein the campaign data comprises a set of content links, atarget cost per click for the set of content links, and a budget amountselected by the first content provider; create an allocation plancomprising assignments of the set of content links for display in aplurality of documents, wherein the allocation plan comprises a targetnumber of impressions for the set of content links; collect useractivity data associated with a first user; determine a firstuser-specific selection probability corresponding to a first contentlink of the set of content links based on the user activity dataassociated with the first user, wherein the first user-specificselection probability represents a calculated probability of servingcontent in response to an interaction by the first user with the firstcontent link; assign, to the first content link, a first graderepresenting a product of the first user-specific selection probabilityassociated with the first user and the first content link and the targetcost per click selected by the first content provider; determine pricingfor provisioning the first content link to one or more of the pluralityof documents; generate a plurality of sub-categories to classify theplurality of documents; determine, using a first prediction model, apredicted number of views for a first document of the plurality ofdocuments, wherein the predicted number of views is based on an averagenumber of views of a first set of documents in a first sub-categorycomprising the first document; allocate, by a processing device, thefirst content link to the first document based on the first grade, theallocation plan, the pricing, the predicted number of views, and thefirst user-specific selection probability; and in response to the firstuser requesting the first document, cause, based on allocating the firstcontent link to the first document, the first content link to be servedvia a designated portion of the first document delivered to the firstuser.
 7. The system of claim 6, wherein the processing device executesthe instructions to: generate a ranking of a plurality ofrecommendations based on a plurality of grades; and provide the rankingto a distribution system to serve the plurality of recommendations toone or more servers providing the plurality of documents.
 8. The systemof claim 7, the processing device to execute the instructions to:determine a first lottery probability of the first content link beingselected in a lottery based on the first grade, wherein the allocatingis further based on the first lottery probability.
 9. The system ofclaim 8, the processing device to execute the instructions to: determinea decrease to the first lottery probability for the first content link;and determine an adjusted price for provisioning of the first contentlink based on the decrease to the first lottery probability.
 10. Anon-transitory computer readable storage medium e comprisinginstructions that, in response to execution by a processing device,cause the processing device to perform operations comprising: receivingcampaign data from a first content provider, wherein the campaign datacomprises a set of content links, a target cost per click for the set ofcontent links, and a budget amount selected by the first contentprovider; creating an allocation plan comprising assignments of the setof content links for display in a plurality of documents, wherein theallocation plan comprises a target number of impressions for the set ofcontent links; collecting user activity data associated with a firstuser; determining a first user-specific selection probabilitycorresponding to a first content link of the set of content links basedon the user activity data associated with the first user, wherein thefirst user-specific selection probability represents a calculatedprobability of serving content in response to an interaction by thefirst user with the first content link; assigning, to the first contentlink, a first grade representing a product of the first user-specificselection probability associated with the first user and the firstcontent link and the target cost per click selected by the first contentprovider; determining pricing for provisioning the first content link toone or more of the plurality of documents; generating a plurality ofsub-categories to classify the plurality of documents; determining,using a first prediction model, a predicted number of views for a firstdocument of the plurality of documents, wherein the predicted number ofviews is based on an average number of views of a first set of documentsin a first sub-category comprising the first document; allocating, by aprocessing device, the first content link to the first document based onthe first grade, the allocation plan, the pricing, the predicted numberof views, and the first user-specific selection probability; and inresponse to the first user requesting the first document, causing, basedon the allocating, the first content link to be served via a designatedportion of the first document delivered to the first user.
 11. Thenon-transitory computer readable storage medium of claim 10, theoperations further comprising: generating a ranking of a plurality ofrecommendations based on a plurality of grades; and providing theranking to a distribution system to serve the plurality ofrecommendations to one or more servers providing the plurality ofdocuments.